Facebook v2 working modules for facebook posts/threads and comments - #95

Hi @uree , this looks very cool! As we discussed a while ago Facebook support is a popular feature request so we would be very happy to merge this. With some quick testing, it seems to work well for posts, though not all comments I'm seeing seem to be captured.

But in any case I think we would be eager to go for Option 3. The challenge being the mapping code (i.e. how to get something useful out of the ndjsons). Looks like you did most of the heavy lifting there too as well, which is very nice.

We recently made some changes to Zeeschuimer to add CSV download functionality to the extension itself (see #77 and #79). So ideally zeextractor's code would be integrated directly and then people would be able to download Facebook CSVs from the extension itself.

The challenge being that zeextractor is Python and Zeeschuimer is JavaScript. But this actually fits our workflow perfectly - all the mapping code is in fact part of 4CAT (which is written in Python), and we semi-automatically translate it to JavaScript for the CSV download feature. This has worked well for the other platforms' modules and having taken a quick look at zeextractor's code I think it should be possible for that as well.

But it would require the mapping code to be in 4CAT directly rather than in an external dependency (see TikTok's mapper function for an example). So I suppose the question is what you think about that. I don't know for example how you use zeextractor in your workflow and how important it is for it to be a standalone app/library. Alternatively, there could be separate codebases (zeextractor and 4CAT).

Very good. zeextractor was written purely for the purpose of processing Zeeschuimer data and it does not seem like it will have a life of its own beyond that, so we can strip it for parts and integrate it with 4CAT. I suppose there would also be a bit less friction when it comes to maintenance if it's all in the same repo?

On the other hand, the quickest path to implementation I see involves using zeextractor as a package. I could quickly take this pipeline out of the loop and into a separate function which returns a dict and we can just install it and call that from SearchFacebook map_item, I assume?

I wrote the mapping based on what we thought was useful for our purposes, but there might be more useful data points in there. That, however, is a can which can safely be kicked down the road, I feel.

About the comments ... Unfortunately, to save all the comments under a post (not just the ones which are visible), you have to click on the comment thread while you scroll. Hovering over the "show more comments" link also worked at some point. It's a legit question whether this is good enough for inclusion (and just add a disclaimer), or should the comment gathering functionality be left out of Zeeschuimer for now. Until someone figures out a way to trigger these "fetch comments" events programmatically, for example.